Unrolled Iterative Estimation
نویسندگان
چکیده
The past year saw the introduction of new architectures such as Highway networks (Srivastava et al., 2015a) and Residual networks (He et al., 2015) which, for the first time, enabled the training of feedforward networks with dozens to hundreds of layers using simple gradient descent. While depth of representation has been posited as a primary reason for their success, there are indications that these architectures defy a popular view of deep learning as a hierarchical computation of increasingly abstract features at each layer. In this report, we argue that this view is incomplete and does not adequately explain several recent findings. We propose an alternative viewpoint based on unrolled iterative estimation—a group of successive layers iteratively refine their estimates of the same features instead of computing an entirely new representation. We demonstrate that this viewpoint directly leads to the construction of Highway and Residual networks. Finally we provide preliminary experiments to discuss the similarities and differences between the two architectures.
منابع مشابه
Highway and Residual Networks learn Unrolled Iterative Estimation
The past year saw the introduction of new architectures such as Highway networks (Srivastava et al., 2015a) and Residual networks (He et al., 2015) which, for the first time, enabled the training of feedforward networks with dozens to hundreds of layers using simple gradient descent. While depth of representation has been posited as a primary reason for their success, there are indications that...
متن کاملA simple RNN-plus-highway network for statistical parametric speech synthesis
In this report, we proposes a neural network structure that combines a recurrent neural network (RNN) and a deep highway network. Compared with the highway RNN structures proposed in other studies, the one proposed in this study is simpler since it only concatenates a highway network after a pre-trained RNN. The main idea is to use the ‘iterative unrolled estimation’ of a highway network to fin...
متن کاملUnrolled Optimization with Deep Priors
A broad class of problems at the core of computational imaging, sensing, and low-level computer vision reduces to the inverse problem of extracting latent images that follow a prior distribution, from measurements taken under a known physical image formation model. Traditionally, hand-crafted priors along with iterative optimization methods have been used to solve such problems. In this paper w...
متن کاملITERATIVE METHOD FOR SOLVING TWO-DIMENSIONAL NONLINEAR FUZZY INTEGRAL EQUATIONS USING FUZZY BIVARIATE BLOCK-PULSE FUNCTIONS WITH ERROR ESTIMATION
In this paper, we propose an iterative procedure based on two dimensionalfuzzy block-pulse functions for solving nonlinear fuzzy Fredholm integralequations of the second kind. The error estimation and numerical stabilityof the proposed method are given in terms of supplementary Lipschitz condition.Finally, illustrative examples are included in order to demonstrate the accuracyand convergence of...
متن کامل